Evaluating DETECT Classification Accuracy and Consistency when Data Display Complex Structure

DETECT is an innovative and relatively new nonparametric dimensionality assessment procedure used to identify mutually exclusive, dimensionally homogeneous clusters of items using a genetic algorithm (Zhang & Stout, 1999). Because the clusters of items are mutually exclusive, this procedure is most useful when the data display approximate simple structure. In many testing situations, however, data display a complex multidimensional structure. The purpose of the current study was to evaluate DETECT item classification accuracy and consistency when the data display different degrees of complex structure using both simulated and real data analyses. Three variables were manipulated in the simulation study: The percentage of items displaying complex structure (10%, 30%, 50%), the correlation between dimensions (0.00, 0.30, 0.60, 0.75, 0.90), and the sample size (500, 1000, 1500). The results from the simulation study reveal that DETECT can accurately and consistently cluster items according to their true underlying dimension when as many as 30% of the items display complex structure, if the correlation between dimensions is less than or equal to 0.75 and the sample size is at least 1000 examinees. If 50% of the items display complex structure, then the correlation between dimensions should be less than or equal to 0.60 and the sample size be, at least, 1000 examinees. When the correlation between dimensions is 0.90, DETECT does not work well with any complex dimensional structure or sample size. These outcomes are further illustrated in two real data analyses. Implications for practice and directions for future research are discussed. DETECT Classification 3 Evaluating DETECT Classification Accuracy and Consistency when Data Display Complex Structure Dimensionality assessment is typically used to identify distinct clusters of items that, when considered collectively, help characterize the constructs measured by a set of test items. Further, dimensionality assessment is intended to help the researcher and practitioner link substantive interpretations with statistical outcomes in order to better understand the examinee-by-item interaction. With most exploratory dimensionality analyses, statistical indices and summaries are first produced to describe the underlying dimensional structure of the data. This statistical information is then interpreted substantively so that succinct terms, such as “scientific reasoning” or “algebra problem solving”, can be used to characterize the dimensions measured by a set of test items for a specific group of examinees. Thus, dimensionality assessment provides one method for connecting complex, substantively-based, test performance with statistical modeling techniques, which are designed to quantify this performance so it can be interpreted and understood across a large sample of examinees. DETECT, the acronym for Dimensionality Evaluation To Enumerate Contributing Traits, is an innovative and relatively new nonparametric dimensionality assessment procedure (Kim, 1994; Zhang & Stout, 1999). It yields different types of quantitative summaries that can be used to link substantive and statistical dimensionality analyses. For example, DETECT identifies the total number of dominant dimensions underlying student performance on a set of test items; it estimates effect sizes to describe the amount of multidimensionality in a set of test items (i.e., Max D index) as well as the nature of the latent structure for these items (i.e., ), and; it specifies which single dimension is measured best by each test item. MAX r To achieve these outcomes, DETECT identifies mutually exclusive, dimensionally homogeneous clusters of items using a genetic algorithm. Because the clusters of items are mutually exclusive, this procedure is most useful when approximate simple structure prevails in the test data (Ackerman, Gierl, & Walker, 2003; Stout, Habing, Douglas, Kim, Roussos, & Zhang, 1996; Zhang & Stout, 1999). To specify these clusters, DETECT attempts to maximize the value of the DETECT index, D(P). This index quantifies the degree of multidimensionality present in P. DETECT Classification 4 The DETECT index is created by computing all item covariances after conditioning on the examinees’ scores using the remaining items. That is, 1 2 ( ) [ ( , )] ( 1) ij i j TT i j N D P E Cov X X n n δ θ ≤ ≤ ≤ = Θ − ∑ = , where is the number of dichotomous items on a test, denotes the partitioning of items into clusters, is the test composite, and n P n k TT Θ i X j X are scores on items i and j , and 1 , 1 . ij if items i and j are in the same cluster of P otherwise δ ⎧ ⎪⎪ = ⎨− ⎪ ⎪⎩ Although many different partitions can exist in a set of test data, P* serves as the partition that maximizes D(P) [herein denoted as D(P*), but also called Max D in the literature]. For instance, when the data are unidimensional, clusters of items will be found that are not homogeneous. In this case, the within-cluster conditional covariance will be positive for some pairs of items and negative for other pairs of items resulting in a D(P*) index that is close to zero. If, however, the data are multidimensional, then clusters of items will be found that have positive within-cluster conditional covariance and negative between-cluster conditional covariance, resulting in a D(P*) index that is greater than zero. Another index that is often reported with D(P*) is . To determine if the partition P*, which produced D(P*) is, in fact, the correct partition to produce a simple structure solution, the following ratio can be computed MAX r